This article examines the reliability issues of large language models (LLMs) used in AI, highlighting their tendency to hallucinate and produce incorrect information. New research indicates that these problems stem from the models' inherent design, raising concerns about their suitability for high-stakes applications like law and accounting. Investors may need to reconsider the viability of AI business models given these risks.

• Hallucination rates rise sharply with input length: GLM 4.5 went from 1.2% errors at 32K words to 3.2% at 128K, and some models hallucinated in most cases at 200K words.
• Tsinghua research suggests hallucination-causing neurons are baked in during initial training, making the flaw structural rather than a fixable bug.
• NYT investigation found LLMs generating tax form errors serious enough to risk legal consequences like tax evasion, casting doubt on their use in high-stakes fields like law and accounting.
• Fixes for hallucination are being explored but likely years away, undercutting near-term AI business models built on high-reliability use cases.